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Abstract 

Research is conducted to cumulate knowledge across studies, so 
assessing the generalizability of results is an essential component 
of scientific inquiry. Unfortunately, statistical significance 
testing does not inform judgment regarding the probability that 
results will replicate. The present paper presents a series of 
related logics that can be enplov^d to empirically evaluate the 
replicability of results. Data reported by Holzinger and Swineford 
(1939), widely available to researchers and frequently used in 
previous heuristic examples, are employed to make the discussion 
concrete. 
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Hinkle, Wiersma and Jurs (1979, p. 415) noted that "it is 
becoming increasingly important for behavioral scientists to 
understand multivariate procedures even if they do not use them in 
their own research." Recent empirical studies of research practice 
confirm that multivariate methods are employed with some regularity 
in behavioral research (Elmore & Woehlke, 1988). Fish (1988) notes 
that there are two reasons why multivariate methods are so 
important in behavioral research. 

^multivariate mpthods limit th^ inflation of Typp t 
^lexaerimentwise" «.rror r^fpg. The seriousness of "experimentwise" 
error inflation, and what to do about it, are both matters 
prompting some disagreement (e.g., Bray & Maxwell, 1982, p. 343, 
1985, p. 10; Hummel & Johnston, 1986). But it is clear that, 
"Whenever multiple statistical tests are carried out in inferential 
data analysis, there is a potential problem of 'probability 
pyramiding'" (Huberty & Morris, 1989, p. 306). And as Morrow and 
Frankiewicz (1979) emphasize, it is also clear that in some cases 
inflation of experimentwise error rates can be quite serious. 

Most researchers are familiar with "testwise" alpha. But while 
"testwise" alpha refers to the probability of making a Type I error 
^ given hypothesis test , "experimentwise" error rate refers to 
the probability of having made a Type I error anywhere wSth.'n 
study, i.e., across all hypotheses. Therefore, when only one 
hypothesis is tested for a given group of people in a study, 
"experimentwise" error rate will exactly equal the "testwise" error 



rate, 
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But when more than one hypothesis is tested in a given study, 
the two error rates will not be equal. Witte (1985, p. 236) 
explains the two error rates using an intuitively appealing example 
involvin>g a coin toss. If the toss of heads is equated with a Type 
I error, and if a coin is tossed only once, then the probability of 
a head on the one toss and of at least one head within the set of 
one toss will both equal 50%. But if the coin is tossed three 
times, even though the "testwise" probability of a head on each 
toss is 50%, the "experimentwise" probability that there will be at 
least one head in the whole set of three flips will be inflated to 
87.5%. This dynamic is illustrated in Table 1. Analogies for 
research are presented in Table 2. Researchers control "testwise" 
error rate by picking small values, usually 0.05, for the 
"testwise" alpha. "Experimentwise" error rate can be limited by 
employing multivariate statistics. 

INSERT TABLES 1 AND 2 ABOUT HERE. 

Paradoxically, although '.he use of several univariate tests in 
a single study can lead to too many null hypotheses being 
spuriously rejected, as reflected in inflation of "experimentwise" 
error rate, it is also possible that the failure to employ 
multivariate methods can lead to a failure to identify 
statistically significant results which actually exist. Fish (1988) 
and Maxwell (1991) both provide data sets illustrating this equally 
disturbing possibility. Thus, "correcting" the testwise alpha level 
(e.g., with a Bonferroni correction — Huberty, 1987) so as to 
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control experimentwise error rate inflation is not a satisfactory 
solution to this problem. The basis for this paradox is beyond the 
scope of the present treatment, but involves the second major 
reason why multivariate statistics are so important. 

Multivariate methods are often vital in behavioral research 
because multivariate methods best honor the reality to which the 
researcher is purportedly trying to generalize . This is 
particularly important, since significance testing and error rates 
may not always be the most important aspect of research practice 
(Thompson, 1989b). Thompson (1986, p. 9) notes that the reality 
about which most researchers wish to generalize is usually one "in 
which the researcher cares about multiple outcomes, in which most 
outcomes have multiple causes, and in which most causes have 
multiple effects." Tatsuoka's (1973, p. 273) previous remarks 
remain telling; 

The often-heard argument, "I'm more interested in 
seeing how each variable, in its own right, affects 
the outcome" overlooks the fact that any variable 
taken in isolation may affect the criterion 
differently from the way it will act in the company 
of other variables. It also overlooks the fact that 
multivariate analysis — precisely by considering all 
the variables simultaneously — can throw light on how 
each one contributes to the relation. 
Although multivariate methods have enjoyed fairly widespread 
usage (Thompson, 1989a; Wood & Erskine, 1976) since computers and 



statistical software became widely available, multivariate methods 
also have been used in intriguing ways in measurement and 
assessment contexts. For example, Merenda, Novack and Bona venture 
(1976) reported a multivariate reliability analysis involving 
subtest scores from the California Test of Mental Maturity. 
Similarly, Sexton, McLean, Boyd, Thompson and McCormick (1988) 
reported results involving a multivariate concurrent validity 
analysis . 

Unfortunately, as Nunnally (1978, p. 298) notes, "one tends to 
take advantage of chance in any situation where something is 
optimized from the data at hand." In fact, this capitalization 
occurs in all classical parametric methods, because all these 
methods (e.g., t-tests, ANOVA, regression, MANOVA) are least 
squares procedures that implicitly or explicitly (a) use weights, 
(b) focus on latent synthetic variables, and (c) yield effect sizes 
analogous to r^, i.e., all classical analytic methods are 
correlational (Knapp, 1978; Thompson, 1988a). 

The problem of capitalizing on sampling error when 
multivariate methods are used is particularly acute, because the 
models being tested involve a larger system of parameter estimates. 
For example, the problem is particularly difficult when factor 
analytic methods are employed, because "one has numerous 
possibilities for capitalizing on chance. Most extraction 
procedures, including principal factor solutions, reach their 
criterion by such capitalization. The same is true of rotational 
procedures, including those which rotate for simple structure" 



(Gorsuch, 1983, p. 330) . 

Thus, it is critically important to evaluate the influences of 
sampling error on obtained results, i.e., the replicability or the 
invar iance of results. Contrary to somewhat common misconceptions, 
gtaUstjc^^l significance testing does not ip f orm -iudg THi^ Tlt reaardir^g 
the prpbablg rep l icability or the sampi ing -^po^^ f ^^ ^ ^,v pf r^s.ni ^y 
(Carver, 1978; Thompson, 1987, 1989b). The purpose of the present 
paper is to describe selected methods for empirically evaluating 
the stability of results, especially as regards multivariate 
analyses. 

The data reported by Holzinger and Swineford (1939, pp. 81- 
91), used with some frequency to illustrate multivariate 
statistical analyses (e.g., Gorsuch, 1983, passim; Joreskog & 
S6rbom, 1986, pp. Ill . 106-III . 122) , are used here to make the 
discussion more concrete. These data were selected for use in the 
examples because they are widely available, and interested readers 
can therefore readily replicate the analyses described here. 
Appendix A presents the SPSS-X program used to generate the 
results. Table 3 presents descriptive statistics and labels for the 
variables. 



INSERT TABLE 3 ABOUT HERE. 



A Univariate Analysis as a Starting Point 
Readers more familiar with univariate analyses may appreciate 
an initial example of a univariate invariance analysis, prior to 
discussion of some methods that can be employed in the multivariate 



case. In a seminal article, Cohen (1968, p. 426) noted that ANOVA 
and ANCOVA are special cases of multiple regression analysis, and 
argued that in this realization "lie possibilities for more 
relevant and therefore more powerful exploitation of research 
data.'* Thus, regression analysis provides a good context for an 
invariance analysis example, because regression is so useful. 

Researchers have increasingly recognized that conventional 
multiple regression analysis of data as they were initially 
collected (no conversion of intervally scaled independent variables 
into dichotomies or trichotomies) does not discard information or 
distort reality, and that the general linear model 

...can be used equally well in experimental or non- 
experimental research. It can handle continuous and 
categorical variables. It can handle two, three, 
four, or more independent variables... Finally, as 
we will abundantly show, multiple regression 
analysis can do anything the analysis of variance 
does — sums of squares, mean squares, F ratios — and 
more. (Kerlinger & Pedhazur, 1973, p. 3) 
Discarding variance is not generally good research practice 
(Kerlinger, 1986, p. 558; Thompson, 1988b) and amounts to 
"squandering of information" (Cohen, 1968, p. 441). As Pedhazur 
(1982, pp. 452-453) notes. 

Categorization of attribute variables is all too 
frequently resorted to in the social sciences... It 
is possible that some of the conflicting evidence in 



the research literature of a given area may be 
attributed to the practice of categorization of 
continuous variables... Categorization leads to a 
loss of information, and consequently to a less 
sensitive analysis. 
One reason why researchc i may be prone to categorizing 
continuous variables is that some researchers unconsciously and 
erroneously associate ANOVA with the power of experimental designs. 
Humphreys (1978, p. 873) notes that: 

The basic fact is that a measure of individual 
differences is not an independent variable, and it 
does not become one by categorizing the scores and 
treating the categories as if they defined a 
variable under experimental control in a factorially 
designed analysis of variance. 
Similarly, Humphreys and Fleishman (1974, p. 468) note that 
categorizing variables in a nonexperimental design using an ANOVA 
analysis "not infrequently produces in both the investigator and 
his audience the illusion that he has experimental control over the 
independent variable. Nothing could be more wrong." 

As Cliff (1987, p. 130) notes, the practice of discarding 
variance on intervally scaled predictor variables to perforra OVA 
analyses creates problems in almost all cases: 

Such divisions are not infallible; think of the 
persons near the borders. Some who should be highs 
are actually classified as lows, and vice versa. In 
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addition, the "barely highs" are classilied the same 
as the "very highs," even though they are different. 
Therefore, reducing a reliable variable to a 
dichotomy makes the variable more unreliable, not 
less. 

These various realizations have led to less frequent use of OVA 
iiethods, and to more frequent use of general linear model 
approaches such as regression (Elmore & Woehlke, 1988; Goodwin & 
Goodwin, 1985; Willson, 1982). 
Example /I: Multiple Regression Analysis 

The regression example utilized scores of the 301 subjects on 
variable T6, paragraph comprehension, as the dependent variable. 
Scores on variables T12, T13, T23, and T24, described in Table 3, 
were arbitrarily selected as the predictors for the example. The 
effect size in the example was noteworthy (R^ = .277) and 
statistically significant (f = 28.395, df - 4/296, p < .0001). 

Researchers have increasingly recognized, however, that the 
effect sizes in parametric analyses are subject to "shrinkage", 
i.e., least squares methods for data in hand tend to be 
overestimates of the effects that would be realized by the 
application of the calculated sample weights to data in other 
samples (Fisk, 1991) . One approach to this problem invokes 
statistical correction formulas grounded in certain assumptions 
about the sample and sampling error. For example, one might employ 
Wherry's (1931) correction formula to R^. The Wherry correction can 
be expressed as: 
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r2 - ((1 - r2) * (V / (n - V - 1))). 

when applied to the example results, the corrected population 

estimate is: 

.27731 - ((1 - .27731) * (4 / (301 - 4 - 1))) 
.27731 - ( .72269 * (4 / 296 )) 

.27731 - ( .72269 * .013513 ) 

.27731 - .009766 
.267543. 

However, Stevens (1986, pp. 78-84) incisively implies that 

researchers usually ground their work in empirical findings from 

previous samples ^ and in actual practice usually want their work to 

generalize to future samples in future research rather than to the 

unknowable population. Herzberg (1969) provides a correction for 

this estimate: 

1 - ((n-l)/(n-v-l)) ((n-2)/(n-v-2)) ((n+l)/n) (1-r2) . 

For these data the correction for would be: 

1 - (300 / 296 ) * (299 / 295 ) * (302/301)* (1-. 27731) 
1 - 1.013513 * 1.013559 * 1.003322 * 0.72269 

1 - 1.027256 * 1.003322 * 0.72269 

1 - 1.030668 * 0.72269 

1 - 0.744854 
0.255145, 

a result which even further overcorrects the estimate, and is thus 
still more conservative. 

These sorts of statistical corrections for effect size 
estimates make smaller corrections both as (a) sample effect sizes 
are larger and (b) sample sizes are larger. However, evaluating the 
stability of an effect size is a different proposition than 
evaluating the stability of the least squares weights used to yield 
the identified effect. Since researchers usually consult the system 
of weights as part of their result interpretation (Thompson & 
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Borrello, 1985), the statistical correction formulas in and of 
themselves are not sufficient for evaluating the invariance of 
results. 

Efforts to estimate the sampling specificity of coefficients 
for specific variables are more difficult, or at least more 
tedious, some researchers randomly split their sample data, conduct 
separate analyses for the two subgroups, and then subjectively 
compare the results to determine if they appear to be similar. Two 
points need to be emphasized about such an approach. 

First, such procedures almost always overestimate the 
invariance or generalizability of results, as Thompson (1984, p. 
46) emphasizes. Most researchers work with samples of convenience 
that are homogeneous in several if not many respects, e.g., 
geographic location. The members of the random subgroups, then, 
have more in common with each other than will independent future 
samples drawn by other researchers. This is not said to discourage 
the practice of replicability analysis, but is emphasized only to 
give a context for the interpretation of results. It is always 
better to have an empirical overestimate of result replicability 
than to have merely a dogmatic attachment to the presumption that 
sample results will generalize. 

Second, it is emphasir-d that inferences regarding 
replicability must be made empirically rather than subjectively, 
e.g., net by visually comparing coefficients across two randomly 
identified sample subgroups. Subjective comparisons will not do, 
because two sets of weights that appear to be different may in fact 
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yield quite similar estimates of the synthetic or latent variables 
actually being correlated in all parametric analyses, e.g., the 
synthetic variable Y being correlated with Y to yield R in a 
regression analysis. Rowel 1 (1991) provides a concrete example of 
just such a case. Cliff (1987, pp. 177-178) suggests that such 
cases involve "insensitivity" of the weights to departures from 
least squares constraints. 

Cross-validation is one vehicle for empirically exploring the 
stability of regression weights and resulting effect sizes across 
samples (Huck, Cormier & Bounds, 1974, pp. 159-160> . Rowell (1991) 
and Thompson (1989b) provide examples for the regression case. Of 
course, other empirical methods are available for evaluating result 
replicability, such as the bootstrap (Diaconis & Efron, 1983; 
Lunneborg, 1987) . Thompson and Melancon (1990) provide an example 
of the application of the bootstrap in the regression case. 

In the first step of the cross-validation process the sample 
is divided into two subsamples, usually randomly. However, for the 
purposes of the present heuristic example, the two schools 
represented in the Holzinger and Swineford (1939) data were used as 
the subgroups. 

Next, separate analyses were conducted for the two data sets. 
The effect size for the 156 students at Pasteur School, = .240, 
was statistically significant (F = 11.927, df = 4/151, p < .0001). 
The prediction equation was: 

< = (+.032125 * Zti2) + (-.031195 * Z^ij) + 

( + .383641 * Zt23) ( + -170181 * 2^24) 
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The effect size for the 145 students at Grant-White School, r2 = 
.313, was statistically significant (r - 15.979, ^ = 4/140, p < 
.0001). The prediction equation was: 

2y < = (-.252179 * 2^12) + ( + .215562 * Z^^^) + 

(+.295226 * 2^23) + (+.333045 * 2^34) 
The third step in the analysis requires that the predictor 
variables in each subsample be standardized into z-score form using 
each group's own means and SDs, and then that new Y's be calculated 
for subsample one using group two's weights (called here Y^j) , and 
for subsample two using group one's weights (called here Yj^) . once 
this is done, correlation coefficients are computed among the 
various synthetic variable estimates. For the example data, these 
results are presented in Table 4. 



INSERT TABLE 4 ABOUT HERE. 



The Table 4 invariance coefficients (.8506 and .8613) suggest 
that the subsample results are reasonably invariant. The results 
also illustrate the importance of empirically evaluating 
invariance, since the weights yield reasonably comparable estimates 
of Y, notwithstanding the fact that the beta weights might appear 
different on the basis of subjective inspection (e.g., +.032125 vs 
-.252179, -.031195 vs +.215562). 

The researcher with such results will conclude that the r2 
effect size is relatively stable, that the beta weights fluctuate, 
but that the effect sizes tend to be "insensitive" to these 
fluctuations. Of course, the regression results for the full sample 
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will then provide the final basis for interpretation. The subsample 
results are employed to evaluate result stability, and are not used 
as the basis for interpretation . The results for the full sample 
are used for interpretation, since these results should 
theoretically be the most stable, as a function of sample size. 
Example Invariance Analyses for Two Multivariate Analyses 
The same logic for evaluating invariance can be readily 
generalized to analyses that are multivariate. Factor analysis and 
canonical correlation analysis were the two methods selected as 
examples of ti:is generalization. 
Example f2\ Factor Analysis 

Factor analysis has been closely associated with evaluating 
the construct validity of measures. Nunnally (1978, p. Ill) notes 
that "construct validity has been spoken of as 'trait validity* and 
•factorial validity.'" Gorsuch (1983, pp. 350-351) suggests that 
A prime use of factor analysis has been in the 
development of both the theoretical constructs for an 
area and the operational representatives for the 
theoretical constructs.. . If a theory has clearly defined 
constructs, then scales can be directly built to embody 
those constructs. 
Thus, "factor analysis is intimately involved with questions of 
validity. . . Factor analysis is at the heart of the measurement of 
psychological constructs" (Nunnally, 1978, p. 112). 

Twenty-four variables from the Holzinger and Swineford (1939) 
data, Tl through T24, were employed in this example. A variety of 
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invariance logics can be applied in factor analysis, including 
••best fit" Procrustean rotation across sample splits (Thompson, 
1991), bootstrap factor analysis (Thompson, 1988c), and various 

other methods described and compared by Guadagnoli and Velicer 
(1988). However, the method selected for discussion here is a 

generalization of the regression cross-validation strategy, a 

method familiar to many researchers. 

A variety of procedures can be employed to calculate the 

synthetic variables in factor analysis, called factor scores 

(Thompson, 1983). However, the most common estimation procedure is 

the regression procedure represented by the matrix algebra 

algorithm; 



where 



^NxV *VxF' 



^VxF ~ '^VxV ^ ^VxF' 



and where Z is the set of y z-scores for each of the n subjects, R-^ 
is the inverse of the correlation matrix for the variable set, and 
P is the orthogonal factor pattern/structure matrix. 

The w matrix is related to the beta weights employed to 
estimate the synthetic variable scores in regression. Thus, one 
approach to evaluating invariance of factor analytic results could 
invoke a comparison of the synthetic factor scores derived using W 
matrices across sample splits. 

Table 5 presents the varimax-rotated structure matrix for the 
156 Pasteur students, and Table 6 presents the associated Weight 
matrix for the students. Table 7 presents i:he varimax-rotated 
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structure matrix for the 145 Grant-White students, and Table 8 
presents the associated Weight matrix for the students. 



INSERT TABLES 5 THROUGH 8 ABOUT HERE. 



Appendix A presents the SPSS-X commands employed to compute 
the factor scores for each group involving each group's own data 
and each group's own weights, i.e., variables "fslll" through 
••fsll4" for the 156 Pasteur students, and variables "fs221" through 
"fs224" for the 145 Grant-White students. Appendix A also presents 
the SPSS-X commands employed to compute the factor scores for each 
group involving each group's own data and the other group's 
weights, i.e., variables "fsl21" through ••fsl24" for the 156 
Pasteur students, and variables "fs211" through "fs214" for the 145 
Grant-White students. 

Table 9 presents the invariance coefficients for analysis, and 
these coefficients are bolded. The first set of coefficients for 
the four principal components scores is: .9677, .9447, .9740, and 
.9590. The second set of coefficients is: .9695, .9606, .9565, 
.9633. If such results had occurred across random subgroups of a 
sample, the researcher would doubtless be relatively sanguine about 
the stability of results. 



INSERT TABLE 9 ABOUT HERE. 



Example #3: Canon ical Correlation Analvsis 

Though multiple regression is a useful analytic method, 
canonical correlation analysis, and not regression analysis, is the 
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most general case of the general linear model (Baggaley, 1981, p. 
129; Fornell, 1978, p. 168). in an important article, Knapp (1978, 
p. 410) demonstrated this in some mathematical detail and concluded 
that "virtually all of the commonly encountered tests of 
significance can be treated as sped :1 cases of canonical 
correlation analysis." Thompson (1988a) illustrates how canonical 
correlation analysis can be employed to implement all the 
parametric tests that canonical methods subsume as special cases. 

Thus, canonical correlation analysis is a powerful analytic 
paradigm that can be applied to myriad research problems. The 
method is valuable because it honors the complexity of reality by 
simu^tanepMsly considering all relationships among variables, and 
does not require that intervally scaled predictor variables be 
converted to nominal scale. As Stevens (1986, p. 373, emphasis 
omitted) notes, 

canonical correlation... is appropriate if the wish 
is to parsimoniously describe the number and nature 
of mutually Independent relationships between the 
two [or more] sets [of variables]... Since the 
combinations [of the variables derived through least 
squares weighting] are uncorrelated, one will obtain 
a very nice additive partitioning of the total 
between association. 
Canonical analysis yields an effect size estimate, R^^, that 
is akin to the r2 effect in multiple regression. Like regression, 
which yields both weights (called beta weights) and correlations 
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coefficients (called structure coefficients) between observed 
predictors (e.g., ^112^ and the synthetic variable, Y (Thompson & 
Borrello, 1985) , canonical analysis also yields least squares 
weights (called canonical function coefficients) and structure 
coefficients. The two sets of coefficients have the same function 
and meaning in canonical analysis as they do in regression, and are 
the two primary rivals for evaluating variable importance at the 
function level (Harris, 1989; Kerlinger & Pedhazur, 1973, p. 344; 
Levine, 1977, p. 20, Meredith, 1964, p. 55; Thompson, in press-b) . 

However, though canonical effect sizes tend to be reasonably 
stable across samples (Thompson, 1990) , the individual function 
coefficients and structure coefficients that are an important 
component of the analysis tend to be less stable (Thompson, in 
press-a) , Thus, invar iance analyses are very important in the 
canonical case, and several methods can be utilized (Thompson, 
1984) . The single method illustrated here is in the same genre as 
the previous examples. 

Variables Tl through T4 were related to variables T5 through 
T9 in the present example. Four uncorrelated canonical functions 
were possible in the example, since the smallest variable set 
consisted of four variables. For the full sample the effect size 
(Rq^ = .00720) for the third function was negligible and the 
likelihood ratio for roots three and four was not statistically 
significant, therefore invariance analyses were conducted only for 
the first two functions. Appendix A presents the COMPUTE statements 
with the canonical function weights used to calculate the synthetic 
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variables correlated in the analysis. For example, the synthetic 
criterion composite variable on Function I for the 145 Pasteur 
students using their data and their weights ("CRITlll") was: 
critlll = (.90048 * p^^j) + ( .09741 * p^^j) + 
(.05615 * P2^,3) + (.08840 * p^^^) 
Table 10 presents the "shrunken" effect size coefficients 
derived for the example. For the first subsample, the of .5108 
shrinks to .3091 when group two's weights are applied to group 
one's data, and the second R^, of .2788 shrinks to -.1831. For the 
second subsample, the of .5195 shrinks to .3685 when group one's 
weights are applied to group two's data, and the second R^ of .1679 
shrinks to -.1282. Though the effect sizes for both the two 
functions were relatively similar across the sample splits, the 
weights employed in the analysis were appreciably more comparable 
for the first than for the second function in both subsamples. 



INSERT TABLE 10 ABOUT HERE, 



Summary 

Statistical significance testing does not inform the 
researcher regarding the replicability of results, yet the business 
of science is formulating generalizabla insight. No one study, 
taken singly, establishes the basis for such Insight. As Neale and 
Liebert (1986, p. 290) observe: 

No one study, however shrewdly designed and 
carefully executed, can provide convincing support 
for a causal hypothesis or theoretical statement... 
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Too many possible (if not plausible) confounds, 
limitations on generality, and alternative 
interpretations can be offered for any one 
observation. Moreover, each of the basic methods of 
research (experimental, correlational, and case 
study) and techniques of comparison (within- or 
between-subjects) has intrinsic limitations. How, 
then, does social science theory advance through 
research? The answer is, by collecting a diverse 
body of evidence about any major theoretical 
proposition. 

Evaluating the generalizability of multivariate results to other 
samples of subjects or of variables is a daunting task, but a task 
which the serious scholar can ill-afford to shirk. Science will 
cumulate knowledge only to the extent that idiosyncratic findings 
are recognized as such, and significance testing is not 
particularly useful for making this evaluation. The present paper 
has illustrated the application of a few of the various logics 
available to the researcher who wishes to pursue such 
investigations . 
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Table 1 

All Possible Families of Outcomes 
for a Fair Coin Flipped Three Times 



Flip # 



p of H on 
each Flip 





1 


2 


1 


1. 


T 


: T 


: T 


2. 


H 


: T 


: T 1 


3. 


T : 


: H : 


: T 1 


4. 


T ; 


; T : 


: h 1 


5. 


H : 


i H ; 


! T j 


6. 


H : 


■ T : 


: H 1 


7. 


T : 


H : 


' H 1 


8. 


H : 


H : 


H 1 




50% 


50% 


50% 



p of 1 or Bore H's (TW error analog) 
in set of 3 Flips = 7/8 = 87.5% 



or 



where TW error analog 
EW p = 1 - n - .5)3 
= I-.53 = 1-.125 



.50, 
.875 



Mote. The probability of one or more occurrences of a given outcome 
in a set of events is 1 - (1-p)*, where fi is the probability of the 
given occurrence on each trial and Jj is the number of trials in a 
set of perfectly independent events. 

Table 2 

Formula for Estimating Experimentwise Type I Error Inflation 
When Hypotheses are Perfectly Uncorrelated 



TW 

alpha 
1 - ( 1 - 0.05 ) * 
1 - ( 0.95 ) * 

1 - 0.95 

Range Over Testwise 
1 - ( 1 - 0.01 ) * 
1 - ( 1 - 0.01 ) * 
1 - ( 1 - 0.01 ) * 
Range Over Testwise 
1 - ( 1 - 0.05 ) * 
1 - ( 1 - 0.05 ) * 
1 - ( 1 - 0.05 ) * 

Range Over Testwise 
1 - ( 1 - 0.10 ) * 
1 - ( 1 - 0.10 ) * 
1 - ( 1 - 0.10 ) * 



Exper imentw i s e 
Tests alpha 

* 1 = 

* 1 - a 

0.05000 
(TW) alpha = .01 



* 5 = 

* 10 

* 20 = 
(TW) alpha 

5 = 
r* 10 = 
r* 20 = 
(TW) alpha 

r* 10 = 
'* 20 - 



0.04901 
0.09562 
0.18209 
.05 

0.22622 
0.40126 
0.64151 
• .10 
0.40951 
0.65132 
0.87842 



ypte. »•**" = "raise to the power of". 

l™nioL«^';^if''^^V°Jl^ /""^ presented (a) to illustrate the 
il^fi V^'' °^ formula step by step and (b) to demonstrate 

that when only one test is conducted, the experimentwise error rate 
VSt ^ w t ^^stwise error rate, as should be expected if the 
formula behaves properly. 
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Table 3 

Descriptive Statistics for Holzinger and Swineford (1939) Data 



Variable Mean 




SD 


Variable Label 




SCHOOL 


1 . 482 




.500 






Tl 


29 « 615 


7 


.005 


VISUAL PERCEPTION TEST FROM SPEARMAN 


VPT 


T2 


24 * 352 


4 


.710 


CUBES, SIMPLIFICATION OF BRIGHAM'S SPATI 


T3 


14 . 229 


2 


.830 


PAPER FORM BOARD—SHAPES THAT CAN BE 


; COM 


T4 


18 . 003 


9 


.048 


LOZENGES FROM THORNDIKE — SHAPES FLIPPED 


T5 


40 . 591 


12 


.381 


'GENERAL INFORMATION VERBAL TEST 




T6 


9 . 183 


3 


.492 


PARAGRAPH COMPREHENSION TEST 




T7 


17 . 362 


5 


. 162 


SENTENCE COMPLETION TEST 




T8 


26 . 126 


5 


.675 


WORD CLASSIFICATION— WHICH WORD NOT 


BELO 


T9 


15 . 299 


7 


.669 


WORD MEANING TEST 




TIO 

A 4,. V 


96 . 276 


25.059 


SPEEDED ADDITION TEST 




Til 


69 . 163 


15 


.670 


SPEEDED CODE TEST—TRANSFORM SHAPES 


INTO 


T12 

A ah M 


110 . 542 


20 


.252 


SPEEDED COUNTING OF DOTS IN SHAPE 




T13 

A A V 


193 . 468 


36 


.329 


SPEEDED DISCRIM STRAIGHT AND CURVED 


CAPS 


T14 


175. 153 


11 


.508 


MEMORY OF TARGET WORDS 






OA 010 


7 


.729 


MEMORY OF TARGET NUMBERS 




T16 


102.525 


7 


.633 


MEMORY OF TARGET SHAPES 




T17 


8.233 


4 


.916 


MEMORY OF OBJECT-NUMBER ASSOCIATION 


TARG 


T18 


9.425 


4 


.488 


MEMORY OF NUMBER-OBJECT ASSOCIATION 


TARG 


T19 


14.037 


4 


.077 


MEMORY OF FIGURE-WORD ASSOCIATION TARGET 


T20 


26.890 


19 


.334 


DEDUCTIVE MATH ABILITY 




T21 


14.249 


4 


.562 


MATH NUMBER PUZZLES 




T22 


26.239 


9 


.197 


MATH WORD PROBLEM REASONING 




T23 


18.136 


9 


.140 


COMPLETION OF A MATH NUMBER SERIES 




T24 


24.266 


4 


.735 


WOODY-MCCALL MIXED MATH FUNDAMENTALS 


TES 


T25 


15.648 


3 


.086 


REVISION OF T3— PAPER FORM BOARD 




T26 


36.303 


8 


.339 


FLAGS— POSSIBLE SUBSTITUTE FOR T4 LOZENG 
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Table 4 

Invariance Coefficients for the Regression Example 

Y Joooo ^" ^" 

a 

.4900 1.0000 
A c e 

Yjj .4168 .8506 1.0000 
d 

Yji .4822 . . 1.0000 

A b f 

Y22 .5599 . . .8613 1.0000 

b^lltf multiple R for the 156 students at Pasteur. 

4m« \% Vlf^""" K^^^v "5 students at Grant-White. 

Pasteup ^LJh ''^}:^^ °^ ^ 156 students at 

s?ud«n^; ^tl K "v"^^"? ^^^^ weights for the Grant-White 

students. The shrinkage is .4900^ - .41682 « .2401 - 1737 - atifiA 

G^ant'^ite" the R for* theS45'ltud"ents ai 

Grant-White, based on using the beta weights for the Pasteur 

4h?f ^^^^^^^^^^ -55992 - .48222 - ^135 - 232I - OS^o! 

Pasteui invariance coefficient for the 156 students at 

^White!^ invariance coefficient for the 145 students at Grant- 
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Table 5 

Varimax Rotated Pattern/Structure Matrix for Pasteur Students 

(n = 156) 



varxaoie 


FACTOR 1 


FACTOR 2 


FACTOR 3 


FACTOR 4 


11 


0 35496 


• 62024 


• 08623 


• 13566 


T2 


• 01632 


. 64395 


-•06319 


- . 06295 


IJ 


• 09557 


• 55294 


• 05815 


24556 


T4 


A ^ A O n 

— . 01989 


*1 A A O 

• 72448 


• 14223 


4 A ^ A M 

. 19744 


1 !> 


.84834 


. 01298 


4 A ^ A A 

• 14422 


- . 07502 


1 O 


• 83010 


• 13268 


4 A A 4 ^ 

« 13814 


. 09395 


1 / 


• 89409 


• 06204 


A ^ A P A 

• 06757 


-. 01908 


To 


• 75202 


• 15203 


• 08439 


. 14611 


T9 


. 80579 


• 22468 


. 17590 


. 04170 


TIO 


• 16125 


- .21005 


A A A A 

. 75728 


. 11505 


Til 


• 35796 


. 03703 


. 62533 


.20684 


112 


. 03551 


• 08427 


.71847 


-. 05683 


Til 1 
X 1 J 


.01455 


.27829 


^ 4 A A 

. 61539 


. 03179 


T14 


.03530 


.05151 


.00034 


.74889 


T15 


-.14516 


.15742 


-.08054 


.71260 


T16 


.08347 


.41132 


.26089 


.50423 


T17 


.05943 


-.09097 


.33372 


.59274 


T18 


.14625 


-.06880 


.03472 


.61530 


T19 


.06763 


.22600 


.27591 


.43278 


T20 


.11427 


,60330 


.04047 


.22728 


T21 


.29862 


.^1483 


.43691 


.13297 


T22 


.50750 


.48108 


.16256 


.01041 


T23 


.39696 


.f>8608 


.24882 


.15038 


T24 


.25074 


.27372 


.53294 


.23170 
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Table 6 

Regression Factor Score Weight Matrix for Pasteur Students 

(n » 156) 



Variable 



FACTOR 1 



FACTOR 2 



FACTOR 



FACTOR 4 



Tl 


. 03613 


T2 


05198 


T3 


03485 


T4 


-•09427 


T5 


. 23775 


T6 


. 22213 


T7 


• 25641 


T8 


. 20342 


T9 


• 20066 


TIO 


02154 


Til 


• 03008 


T12 


-.08539 


T13 


-.09797 


T14 


.01041 


T15 


-.04524 


T16 


-.04415 


T17 


-.00964 


T18 


.05097 


T19 


-.03277 


T20 


-.02896 


T21 


-.00314 


T22 


.08612 


T23 


.03258 


T24 


-.01461 



1 fifinn 

« X oo v/u 


* . 05855 


.00734 




- . 06732 


-.06730 




. 00877 


-.15560 




* . 00020 


.01818 


. vO <J w X 


— . 02595 


-.04572 




. 05440 


.02127 




* . 07813 


-.01511 


. \/ J ^ 7 


- . 07813 


. 04886 




-♦03323 


-.01324 


. X ^ J 


. 34434 


-.02159 


. v / / D 9 


. 23000 


. 01804 


- . 01425 




11566 


.06293 


.27168 


-.07840 


-.03819 


-.09559 


. 32932 


.02857 


-.11519 


.31381 


.09627 


.02875 


.16406 


-.10607 


.08931 


.23123 


-.08815 


-.06895 


.27537 


.03066 


.05849 


.14391 


.20182 


-.06092 


.05311 


.09159 


.13054 


-.02052 


.12328 


-.01656 


-.04619 


.3 888 


.01494 


-.00248 


.02850 


.18016 


.02203 
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Table 7 

Variroax Rotated Pattern/Structure Matrix for Grant-White Students 

(n « 145) 



Variable 

Tl 

T2 

T3 

T4 

T5 

T6 

T7 

T8 

T9 

TIO 

Til 

T12 

T13 

T14 

T15 

T16 

T17 

T18 

T19 

T20 

T21 

T22 

T23 

T24 



FACTOR 1 


FACTOR 0 




. 18969 






. 09441 




. 02583 


. 16318 


51 71 R 




. 24833 


. 71458 




. 77924 


. 21503 


1 O C C *T 


. 80685 


• X 47 X Q X 


Arr ii n *v 


. 84695 


• X X 7 




. 65732 


■ ^ W O W 




. 84512 


• X O 4b 7 7 


.04717 


. 16110 




. 83716 


. 17916 


1 9677 


rfT A O C T 


. 02526 


. 22914 


•7 D >l "7 ^ 


. 19371 


■ *t ^ ^ J & 


• 51470 


. 20665 


. 02488 


AC C *7 


. 07807 


. 16183 


.01560 


. 07075 


.45477 


. 03646 


. 16902 


-.07154 


.27888 


. 00698 


.35969 


.36721 


. 17820 


. 13186 


.22208 


.45866 


.38635 


.05303 


. 17998 


.48808 


.44788 


.44465 


.36018 


. 10038 


.44543 


.48780 


.20782 


. 41133 


.08011 


. 55939 



FACTOR 4 

.16118 

.01294 

.12246 

.07524 

.07616 

.19990 

.07617 

.11186 

.18076 

.11438 

.33076 

.04228 

.02085 

.66968 

.64922 

.57780 

.70520 

.47717 

.45047 

.33040 

. 18824 

.33810 

.23026 

.28566 
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Regression Factor Score Weight Matrix for Grant-White Students 

(n « 145) 



Variable 

Tl 

T2 

T3 

T4 

T5 

T6 

T7 

T8 

T9 

TIO 

Til 

T12 

T13 

T14 

T15 

T16 

T17 

T18 

T19 

T20 

T21 

T22 

T23 

T24 



FACTOR 1 

-.05942 
-.06284 
-.03719 
-.02332 
.23536 
.25056 
.27651 
.17620 
.26728 
-.00634 
-.03412 
-.08810 
-.04231 
-.00362 
-.06587 
-.10440 
-.02489 
-.12885 
-.01861 
.07507 
-.05779 
.06915 
.05464 
.06746 



FACTOR 2 

.27437 
.30421 
.20040 
.29744 
-.03608 
-.07273 
-.08955 
-.01198 
-.06760 
-.15925 
-.06617 
.02109 
.12665 
-.10132 
-.00971 
.14043 
-.17071 
.08030 
-.03707 
.07865 
.14348 
.06203 
.12554 
-.11178 



FACTOR 3 

-.02782 
-.06587 
-.01860 
-.07279 
-.00039 
-.06840 
-.00928 
.03046 
-.08271 
.37219 
.21552 
33573 
.22256 
-.07867 
-.09831 
-.10957 
.03516 
.07278 
.01905 
-.09283 
.12063 
-.06786 
-.01729 
.18538 



FACTOR 4 

-.03557 
-.08359 
-. 02880 
-.07783 
-.09318 
-.00593 
-. 08556 
-.07281 
-.01840 
-.05343 
.06830 
-.10645 
-.14094 
.33244 
.32409 
.24912 
.33842 
.17023 
.18221 
.07819 
-.03113 
.08093 
-.00647 
.03025 
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Table 9 

Factor Score Correlation Matrix 



FSlli 

FS112 

FS113 

FS114 

FS211 

FS212 

FS213 

FS214 

FS221 

FS222 

FS223 

FS224 

FS121 

FS122 

FS123 

FS124 



FSlll 


FS112 


FS113 


FS114 


1.0000 






(156) 








.0000 


1.0000 






(156) 


(156) 






.0000 


.0000 


1.0000 




(156) 


(156) 


(156) 




. 0000 


.0000 


. 0000 


1. 0000 


{lot) 

* 


(156) 


(156) 


(156) 


( 0) 

• 


* 

( 0) 


• 

( 0) 


• 

( 0) 


/ A % 

( 0) 

* 


• 

( 0) 


( 0) 


• 

( 0) 


( 0) 


• 

( 0) 


• 

( 0) 


• 

( 0) 


( 0) 


*( 0) 


( 0) 


*( 0) 


( 0) 


*( 0) 


'( 0) 


'( 0) 


( 0) 


( 0) 


( 0) 


( 0) 


( 0) 


'( 0) 


( 0) 


'( 0) 



FS211 FS212 rS213 FS214 FS221 FS222 FS223 



( 0) (0) (0) ( 0) 



.9695 

(156) 
-.0520 
(156) 
.0844 
(156) 
.1401 
(156) 



.0626 
(156) 
.9606 

(156) 
-.2062 
(156) 
.0399 
(156) 



.0248 
(156) 
-.0241 
(156) 
.9565 
(156) 
.0234 
(156) 



-.1319 
(156) 

-.0578 
(156) 

-.0161 
(156) 
.9633 
(156) 



1. 0000 
(145) 
.2397 
(145) 
.0474 
(145) 
.2756 
(145) 
.9677 
(145) 
.0747 
(145) 
.0518 
(145) 
.1281 
(145) 

( 0) 

( 0) 

• 

( 0) 
( 0) 



l.OOGO 
(145) 
.0790 
(145) 
.2015 
(145) 
.1956 
(145) 
.9447 
(145) 

-.0497 
(145) 
.1401 
(1.45) 

( 0) 

( 0) 

( 0) 

• 

( 0) 



1.0000 






(145) 






.1247 


1.0000 




(145) 


(145) 




-.0006 


.1380 


1.0000 


(145) 


(145) 
.0734 


(145) 


.1017 


.0000 


(145) 


(145) 


(145) 


.9740 


.0694 


.0000 


(145) 


(145) 


(145) 


.0808 


.9590 


.0000 


(145) 


(145) 


(145) 



( 0) (0) ( 0) 

( 0) (0) ( 0) 

( 0) (0) ( 0) 

( 0) ( 0) ( 0) 



1.0000 
(145) 
.0000 
(145) 
.0000 
(145) 

( 0) 

*( 0) 

( 0) 

'( 0) 



1.0000 
(145) 
.0000 
(145) 

'( 0) 

*( 0) 

( 0) 

'( 0) 
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Table 9 (cont.) 





FS224 


FS121 


FS122 


FS123 


FS124 


FS121 






1.0000 


-.0236 


.0781 


-.2568 




( 


0) 


(156) 


(156) 


(156) 


(156) 


FS122 


« 




-.0236 


1.0000 


-.2031 


-.0436 




( 


0) 


(156) 


(156) 


(156) 


(156) 


FS123 


• 




.0781 


-.2031 


1.0000 


-.0367 




( 


0) 


(156) 


(156) 


(1^6) 


(156) 


FS124 


• 




-.2568 


-.0436 


-.0367 


1.0000 




( 


0) 


(156) 


(156) 


(156) 


(156) 



Note . The two sets of four invariance coefficients are presented in 
bold. 
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Table 10 

Correlation Coefficients for Various Pairs of Canonical Variate Scores 



CRITlll 


CRIT112 


PREDlll 


PRED112 


CRIT121 


CRIT122 


PRED121 


PRE0122 


CRIT211 


CRIT212 


PRED211 




X • UUUU 


























nnno 
V xao; 


X * 


noon 

k UUUU 

r' 1 S61 

^ X ^ v9 y 
























7 


» UUUU 


1, 0000 




















^ x3o; 


L X ^ w y 




















ItIvCaL/X x^ 


* u u u u 
^ x^o y 


r 
L 




7 .0000 

(156) 


1. 0000 
( 156) 
















^l\X X X ^ X 


« O J o / 

r 1561 




[156) 


• 4274 
(156) 


-.0824 
(156) 


1. 0000 
( 156) 














v-rvX X X 4C 


• U ^ ^ J 

^ 1561 

V x;?o^ 




i 4? / U^ 

f 1561 


.0170 
(156) 


-. 1590 
(156) 


-.0082 
(156) 


1. 0000 
( 156) 












f'KCii/X/: X 


• ft ^ 04 




k UZ^ 7 J 


.8348 


. 3563 


. 3091 


-.0211 


1. 0000 












M *561 




^ X ^ o y 


^1561 


1 156i 


1 156i 


( 156) 


(156) 










r'KCiUX^^ 


* 1 1 ft n 
• xxou 




O A 7 1 


-.2311 


. 8864 


-.1465 


1831 


.0035 


1. 0000 










^ x:7o; 




f 1 561 


M 561 
^ X ^ w y 


1 1561 

^ XoP w y 




( 1561 

^ X<^ V / 


^156^ 


^156^ 








L.KX i Z XX 


• 


















1. 0000 








/ ni 




^ 01 


( 01 


( 01 


( 01 


r 01 


( 01 


( 01 


^ 145i 






L.KX X A X^ 


• 


















-.0310 


1. 0000 










( 01 


I 01 


I 01 


( 01 


( 01 


I 01 


( 01 


r 145) 


fl45) 




PRED211 




















.3685 


-. 1174 


1.0000 




( 0) 




( 0) 


\ 0) 


"( 0) 


*( 0) 


\ 0) 


'( 0) 


*( 0) 


(145) 


(145) 


(145) 


PRED2I2 




















.1632 


-.1282 


.0060 




( 0) 




( 0) 


*( 0) 


*( 0) 


( 0) 


*( 0) 


*( 0) 


*( 0) 


(145) 


(145) 


(145) 


CRIT221 




















.8306 


-.3105 


.4312 




( 0) 




( 0) 


( 0) 


*( 0) 


*( 0) 


'( 0) 


( 0) 


*( 0) 


(145) 


(145) 


(145) 


CRIT222 




















.0769 


-.6145 


-.0328 




( 0) 




'( 0) 


( 0) 


( 0) 


*( 0) 


*( 0) 


*( 0) 


*( 0) 


(145) 


(145) 


(145) 


PRED221 




















.4315 


-.1613 


.8301 




*( 0) 




'( 0) 


*( 0) 


*( 0) 


*( 0) 


*( 0) 


( 0) 


\ 0) 


(145) 


(145) 


(145) 


PRED222 




















.0129 


-.1032 


-.1951 




*( 0) 




*( 0) 


*( 0) 


( 0) 


"( 0) 


'( 0) 


( 0) 


*( 0) 


(145) 


(145) 


(145) 
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Table 10 (cone.) 





PRED212 


CRIT221 


CRIT222 


PRED221 


PRED222 


PRED212 


1.0000 


.1728 


.1492 


.3326 


.8887 




(145) 


(145) 


(145) 


(145) 


(145) 


CRIT221 


.1728 


1.0000 


.0000 


[ .5195 


] .0000 




(145) 


(145) 


(145) 


(145) 


(145) 


CRIT222 


.1492 


.0000 


1.0000 


.0000 


[ .1679 ] 




(145) 


(145) 


(145) 


(145) 


(145) 


PRED221 


.3326 


.5195 


.0000 


1.0000 


.0000 




(145) 


(145) 


(145) 


(145) 


(145) 


PRED222 


.8887 


.0000 


.1679 


.0000 


1.0000 




(145) 


(145) 


(145) 


(145) 


(145) 



Note . The two R^. values for the two subsainples are presented in 
[square brackets in italics]. The two sets of invariance 
coefficients are presented in bold. 



Appendix A: 

SPSS-X Program Used to Implement the Reported Analyses 

TITLE 'Holzinger & Swineford (1939) Data **Citation in Comment** • 

C 0 M M E N T 

*********************************************************************** 

COMMENT Holzinger, K.J., & Swineford, F. (1939). A study in factor analysis: 
COMMENT The stability of a bi-factor solution (No. 48) . Chicago, ILi 

COMMENT University of Chicago, (data on pp. 81-91) 

COMMENT 
*********************************************************************** 

DATA LIST FILE^BT REC0RDS=2 

/I ID 1-3 SEX 4 AGEYR 6-7 AGEMO 8-9 

Tl 11-12 T2 14-15 T3 17-18 T4 20-21 T5 23-24 T6 26-27 

T7 29-30 T8 32-33 T9 35-36 TlO 38-40 Til 42-44 T12 46-48 

T13 50-52 T14 54-56 T15 58-60 T16 62-64 T17 66-67 

T18 69-70 T19 72-73 T20 74-76 T21 78-79 

/2 T22 11-12 T23 14-15 T24 17-18 

T25 20-21 T26 23-24 
COMPUTE SCHOOL*! 
IF (ID GT 200)SCHOOL=2 
IF (ID GE 1 AND ID LE 85)GRADE=7 
IF (ID GE 86 AND ID LE 168)GRADE=8 
IF (ID GE 201 AND ID LE 281)GRADE=7 
IF (ID GE 282 AND ID LE 351)GRADE=8 
IF (ID GE 1 AND ID LE 44)TRACK=2 
IF (ID GE 45 AND ID LE 85)TRACK=1 
IF (ID GE 86 AND ID LE 129)TRACK-2 
IF (ID GE 130)TRACK==1 
PRINT FORMATS SCHOOL TO TRACK (Fl.O) 
VALUE LABELS SCHOOL ( 1 ) PASTEUR (2) GRANT-WHITE/ 

TRACK (l)JUNE PROMOTIONS (2) FEB PROMOTIONS/ 
VARIABLE LABELS Tl VISUAL PERCEPTION TEST FROM SPEARMAN VPT, PART III 

T2 CUBES, SIMPLIFICATION OF BRIGHAM'S SPATIAL RELATIONS TEST 

T3 PAPER FORM BOARD— SHAPES THAT CAN BE COMBINED TO FORM A TARGET 

T4 LOZENGES FROM THORNDIKE— SHAPES FLIPPED OVER THEN IDENTIFY TARGET 

T5 GENERAL INFORMATION VERBAL TEST 
T6 PARAGRAPH COMPREHENSION TEST 
T7 SENTENCE COMPLETION TEST 

T8 WORD CLASSIFICATION— WHICH WORD NOT BELONG IN SET 
T9 WORD MEANING TEST 

TlO SPEEDED ADDITION TEST 

Til SPEEDED CODE TEST— TRANSFORM SHAPES INTO ALPHA WITH COOE 

T12 SPEEDED COUNTING CF DOTS IN SHAPE 

T13 SPEEDED DI SCRIM STRAIGHT AND CURVED CAPS 

T14 MEMORY OF TARGET WORDS 
T15 MEMORY OF TARGET NUMBERS 
T16 MEMORY OF TARGET SHAPES 

T17 MEMORY OF OBJECT-NUMBER ASSOCIATION TARGETS 
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T18 MEMORY OF NUMBER-OBJECT ASSOCIATION TARGETS 
T19 MEMORY OF FIGURE-WORD ASSOCIATION TARGETS 

T20 DEDUCTIVE MATH ABILITY 

T21 MATH NUMBER PUZZLES 

T22 MATH WORD PROBLEM REASONING 

T23 COMPLETION OF A MATH NUMBER SERIES 

T24 WOODY-MCCALL MIXED MATH FUNDAMENTALS TEST 

T25 REVISION OF T3— PAPER FORM BOARD 

T26 FLAGS — POSSIBLE SUBSTITUTE FOR T4 LOZENGES 
LIST VARIABLES»ALL/CASES=500/FORMAT=NUMBERED 
SUBTITLE '0 DESCRIPTIVES ON ALL SUBJECTS POOLED' 
FREQUENCIES VARIABLES=ID TO T26/FORMAT=NOTABLE/STATISTICS=ALL 
DESCRIPTIVES VARIABLES=SCH0OL Tl TO T26 

SUBTITLE 'la SCH=PASTEUR COMPARE WITH GORSUCH PP. 384-385' 
TEMPORARY 

SELECT IF (SCHOOL EQ 1) 
DESCRIPTIVES VARIABLES=ALL 

SUBTITLE 'lb SCH=GRANT-WH COMPARE WITH GORSUCH PP. 384-385' 
TEMPORARY 

SELECT IF (SCHOOL EQ 2) 
DESCRIPTIVES VARIABLES=ALL 

SUBTITLE *2 COMPARE R MATRIX WITH GORSUCH P. 100' 

CORRELATIONS VARIABLES=T1 TO T26 

SUBTITLE '33 REGRESSION §§§§§§§ ALL CASES' 

REGRESSION VARIABLES=T6 T12 T13 T23 T24/DESCRIPTIVES=MEAN STDDEV CORR/ 

DEPENDENT=T6/ ENTER T12 T13 T23 T24 
SUBTITLE 'Sb REGRESSION 0§§ftit Pasteur' 
TEMPORARY 

SELECr IF (SCHOOL EQ I) 

REGRE^dlON VARIABLES=T6 T12 T13 T23 T24/DESCRIPTIVES==MEAN STDDEV CORR/ 

DEPENDENT»T6/ ENTER T12 T13 T23 T24 
SUBTITLE •3b REGRESSION Grant-White' 
TEMPORARY 

SELECT IF (SCHOOL EQ 2) 

REGRESSION VARIABLES=T6 T12 T13 T23 T24/DESCRIPTIVES=MEAN STDDEV CORR/ 

DEPENDENT-T6/ ENTER T12 T13 T23 T24 
SUBTITLE '48 FACTOR ALL CASES' 

FACTOR VARIABLES=T1 TO T24/PRINT=DEFAULT FSCORE 
SUBTITLE •4b FACTOR Pasteur' 
TEMPORARY 

SELECT IF (SCHOOL EQ 1) 

FACTOR VARIABLES«T1 TO T24/PRINT=DEFAULT FSCORE/CRITERIA= FACTORS (4) 

SUBTITLE '40 FACTOR Grant-White' 

TEMPORARY 

SELECT IF (SCHOOL EQ 2) 

FACTOR VARIABLES=T1 TO T24/PRINT=DEFAULT FSCORE/ CRITERIA=FACTORS (4 ) 
SUBTITLE 'Sa CANONICAL ALL CASES • 

MANOVA Tl TO T4 WITH T5 TO T9/PRINT=SIGNIF(EIGEN DIMENR) / 

DISCRIM(STAN COR ALPHA ( .999) /DESIGN 
SUBTITLE 'Sb CANONICAL Pasteur • 

TEMPORARY 
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SELECT IP (SCHOOL EQ 1) 

^?oiJi.'^ '^^ T9/PRINT=SIGNIF(EIGEN DIMENR) / 

DISCRIM(STAN COR ALPHA (. 999) /DESIGN UAnt.wK; / 

SSS^J^i-f CANONICAL Grant-White' 

TEMPORARY 

SELECT IF (SCHOOL EQ 2) 

"^?ciJi.'^*^ T9/PRINT=SIGNIF(EIGEN DIMENR)/ 

DISCRIM(STAN COR ALPHA (. 999) /DESIGN t-^nt-WK; / 

subtitle 'ea z scores for Pasteur only' 
temporary 

select if (school eq 1) 

'^^tfi''/S^f^f\^*^^^?^^^=^^ ^P^^^J ^2 (pzt2) t3 (pzt3) t4 (pzt4) t5 (pzt5) 
lU^^ ^tL^^ ^P^^^^ ^® fP^^S) ^9 (Pzt9) tlO pztlO) til (pztin ' 

in \llin\ lil l^^l^'^ (P^^^^J (P^tis) tie (pztie) ^ 

III [iiiii] III [iiiii] III mill] 'p--) 

subtitle 'eb z scores for Grant-White only' 
temporary ^ 
select if (school eq 2) 

"^^^^^5^^^?%^^^^^^?^:^' ^2 (g2t2) t3 (g2t3) t4 (gzt4) t5 (gztS) 

t6 (gzt6) t7 (gzt7) t8 (gztS) tP (gzt9) tlO qztlO) til faztli? 

tiv tlini IV. (g°tl5) t?6 (gzt ir''^ 

{|zt^^^^ iii \^ m \iiiii] ^^^^20 t2i (izt2ii 

subtitle '7 regression invariance analysis' 

'TsMSpetH)' ■ - ( . 215562*petl3 , . ( . 295226*p^t23 , . 

'T3M*gm;|-'"'"*'^"''"<-^""^*5^"^'^(-"""*g^t23). 
variable labels 

yhatll 'group 1 data group i weights' 

yhatl2 'group i data group 2 weights' 

I^'''*"P 2 d^ta group 1 weights' 
yhat22 'group 2 data group 2 weights' 

sSbtU?e T fllin^^^^^^^ ^^^^^^ yhat22/3tatistics=descriptiyes 
suDtitle '8 factor score invariance analysis' 

compute fslll= ( . 03613*pztl) - ( . 05198*p2t2) - ( . 03485*pzt3 ) - ( 09427*D2t4 ^ 

+( . 08612*pzt22 ) + ( . 03258*pzt23 - . 01461*pzt24 ' ' ^ 
compute fsll2- ( . 18800*pztl) +( . 25519*pzt2) +( . 21754*Dzt3 1 + r yiTi^*^,^. , 

rno!J2!P^!^^^'*'^'°2857*pztl5) + (.09627*pztl6 - .10607*DZtl7 
- .08815*pztl8)+(.03066*pztl9)4.(.20182*pzt20 + 0915^ 
+( . 12328*pzt22 ) + ( . 15888*pzt23 + . 02850*pzt24 ^ ' °5159*pzt21) 



compute f slX3-=- ( . 05855*pztl) - ( . 06732*pzt2 ) + ( . 00877*p2t3) - ( . 00020*pzt4 ) 
I * ^...^.y^^^^ " ^ • 05440*pzt6) - ( . 07813*pzt7) - ( . 07813*pzte) - ( . 03323*pzt9) 
+(.34434*pztlO)+(.23000*pztlX)+(.34184*pztl2)+(.27168*pztl3) 
'"5'«!!!!!P'^^^*^"^*^^^^^*P2^^5J+(-02875*pztl6)+(.08931*pztl7 
-(. 06895*pztl8) +( . 05849*pztl9) - ( . 06092*pzt20) +( . 13054*pzt21 

•-(.01656*pzt22) + (.01494*pzt23)+(.18016*pzt24) 
^'^^"'^"nT.Jji^^r (•00'734*pztl)-(.06730*pzt2)-(.15560*pzt3) + (.01818*pzt4) 
" • niVdl^^^^i'' ^ • 02127*pzt6) - ( . 01511*pzt7) + ( . 04886*pzt8) - ( . 01324*pzt9) 

- . 02159*pztl0) + ( . 01804 *pztll) - ( . 11566*pztl2) - ( . 07840*pztl3) 

+(.32932*pztl4)+(.31381*pztl5)+(.16406*pztl6)+(.231?3*pztX7 
+(.27537*pztl8)+(.14391*pztl9)+(.05311*pzt20)-(.02052*pzt21) 
-(.04619*pzt22) -( .00248*pzt23)+(.02203*pzt24) 

''°!?''o?^^c?^^rcJ-?^^"*^'=^^^"^-°5^^«*^2t2)-(.03485*g2t3)-(.09427*-^t4) 
rJ^n?!^^^^^'^^-^^^^^*^'^^^J'^(-25641*gzt7)+(.20342*gzt8)+(.20066*g2t9) 

- .02154*g2tl0)+(.03008*gztll)-(.08539*g2tl2)-(.09797*gztl3) 
+ (.01041*gztl4)-(.04524*gzt5.5)-(.04415*gztl6)-(.00964*gztl7) 
+( . 05097*gztl8) - ( . 03277*gztl9) - ( . 02896*gzt20) - ( . 00314*gzt21 
+(. 08612*gzt22) +( . 03258*gzt23) - ( . 01461*gzt24) 

'''''^^t''nt',if*^^ZJ ' ^8800*g7,tl) +( . 25519*gzt2) +( . 21754*gzt3) + ( .25735*gzt4) 
"r?!?!i^^^^^~^-°^^^®*^2^^)"(-°^830*g2t7)-(.03259*gzt8)-(.00839*gzt9) 
- ( . 15523*gztlO) - ( . 07765*gztll) - ( . 01425*gztl2) +( . 06293*gztl3 ) 

-(.03819*gztl4) +(. 02857 *gztl5)+(.09627*gztl6)-(.10607*gztl7) 
-(.08815*g2tl8)+(.03066*gztl9)+(.20182*gzt20)+(.09159*gzt21) 
+(.12328*gzt22)+(.X5888*gzt23)+(.02850*gzt24) 

compute fs213=-(.05855*gztl)-(.06732*gzt2)+(.00877*gzt3)-(.00020*gzt4) 
+ ' JSn^''^^^ " ^ • 07813*gzt7) - ( . 07813*gzt8) - ( . 03323 *gzt9) 

+(.34434*gztl0)+(.23000*g2tll)+(.34184*gztl2)+(.27168*gztl3) 
-(.09559*gztl4)-(.11519*gztl5)+(.02875*gztl6)+(.08931*gztl7 
-(.06895*gztl8)+(.05849*gztl9)-(.06092*gzt20)+(.13054*gzt21) 
- (.01656*gzt22)+(.01494*gzt23)+(.l8016*gzt24) 

compute fs2 14= ( . 00734*gztl) -( . 06730*gzt2) - ( . 15560*gzt3) +( . 01818*gzt4) 

" •n^?Io!^''5fit^;°^^^^*^^^®^"^-°^^^^*9Zt7) + (.04886*gzt8)-(.01324*gzt9 
I "^^ • 01804*gztll) -( . 11566*gztl2) -( . 07840*gztl3) 

+(.32932*gztl4)+(.31381*gztl5)+(.16406*gztl6)+(.23123*gztl7 
{•nI^?«!^^^^®^''^-^^^^^*9^*^^5J + (-0S311*gzt20)-(.02052*gzt21 
-(.04619*gzt22)-(.00248*g2t23)+(.02203*gzt24) 

''°I!!?''o?=f?"^:"^-*^^^'*2*^='^^J"(-0^284*gzt2)-(.03719*gzt3)-(.02332*gzt4) 

' *nnJ2J!^^^^°^"<-°^^^2*gztll).(.08810*gztl2)-(.04231*gztl3) 
~ ; • ~ ^ • 06587*g2tl5) - ( . 10440*gztl6) - ( . 02489*gztl7) 

7 r " ( • °^®^^*gztl9) +( . 07507*gzt20) - ( . 05779*gzt21) 
+(.06915*gzt22)+(.05464*gzt23)+(.06746*g2t24 
compute fs222=« ( . 27437*gztl) +( . 30421*gzt2) +( . 20040*gzt3) +( . 29744*gzt4 ) 
" • VJo^.iy^lV ■ ^ • 07273*gzt6) - ( . 08955*gzt7) - ( . 01198*gzt8 1 (:0676?^gzt9) 
■ •^^??^^^^^°^"^-°^^^^*9^^1^)+^02109*gztl2) + (.12665*gztl3) ^ 
Zi'lolliy^l^^^ " ^ • 00971*g2tl5) + ( . 14043*gztl6) - ( . 17071*g2tl7 

+(.06203*g2t22)+(.12554*gzt23)-(.11178*gzt24 
^'''"'^"^nnJf J^^r ^ • 02782*gztl) - ( . 06587*gzt2) - ( . 01860*gzt3) - ( . 07279*gzt4 ) 

•n3oir^''^^°^''^-21552*gztll) + (.33573*gztl2) + (.22256*gztl3) 
-(.07867*gztl4)-(.09831*g2tl5)-(.10957*g2tl6)+(.03516*gztl7) 
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+ ( . 07278*g2tl8) + ( . 01905*g2tl9) - ( . 09283*g2t20) -»■( . 12063*gzt21) 
-( . 06786*g2t22) - ( . 01729*g2t23) +( . 18538*g2t24) 

compute f s224=- ( . 03557*gztl) - ( . 08359*g2t2) - ( . 02880*g2t3 ) - ( . 07783*gzt4 ) 
- ( . 09318*g2t5) - ( . 00593*gzt6) - ( . 08556*g2t7) - ( . 07281*g2t8) - ( . 01840*g2t9) 
-( .05343*g2tl0) +( . 06830*g2tll) - ( . 10645*g2tl2) - ( . 14094*g2tl3) 
+ (.33244 *g2tl4 )+(. 32409*gztl5) + ( .24912*gztX6)+(.33842*g2tl7) 
+(.17023*g2tl8)+(.18221*g2tl9)+(.07819*g2t20)-(.03113*g2t2l) 
+ ( . 08093*g2t22) -( . 00647*gzt23) + ( . 03025*gzt24) 

compute f sl21=- ( . 05942*p2tl) - ( . 06284*pzt2) - ( . 03719*pzt3) -( . 02332*p2t4 ) 
+( . 23536*p2t5) + ( . 25056*p2t6) + ( . 27651*p2t7) + ( . 17620*p2t8) + ( . 26728*p2t9) 
-(.00634*p2tl0) -(.03412*p2tll)-(.08810*pztl2)-(.04231*p2tl3) 
-(.00362*p2tl4) -(.06587*pztl5)-(.10440*pztl6)-(.02489*p2tl7) 
-( . 12885*pztl8) -( . 01861*pztl9)+( . 07507*pzt2 0) - ( . 05779*p2t21) 
+(.06915*p2t2 2)+(.054 64*pzt23)+(.06746*pzt24) 

compute fsl22= ( . 27437*pztl) + ( . 30421*pzt2) +( . 20040*pzt3) +( . 29744*p2t4 ) 
-(.03608*p2t5)-(.07273*pzt6)-(.08955*p2t7)-(.01198*pzt8)-(.06760*pzt9) 
- ( . 15925*pztl0) -( . 06617*pztll) + ( . 02109*p2tl2) + ( . 12665*pztl3) 
-( . 10132*p2tl4) -( . 00971*pztl5) +( . 1404 3*pztl6) - ( . 17071*pztl7) 
+(.08030*p2tl8)-(.03707*pztl9)+(.07865*pzt20)+(.14348*pzt21) 
+(. 06203 *pzt22 )+(. 12554*pzt23)-(.11178*p2t24) 

compute f sl23=- ( . 02782*pztl) - ( . 06587*pzt2) - ( . 01860*pzt3) - ( . 07279*p2t4) 

-(.00039*pzt5)-(.06840*pzt6)-(.00928*p2t7)+(.03046*p2t8)-(.08271*pzt9) 
+(.37219*pztl0)+(.21552*p2tll)+(.33573*p2tl2)+(.22256*p2tl3) 
-(.07867*p2tl4)-(.09831*pztl5)-(.10957*pztl6)+(.03516*pztl7) 
+ ( . 07278*p2tl8) +( . 01905*pzi-19) - ( . 09283*pzt20) +( . 12063*pzt21) 
-( . 06786*pzt22) - ( . 01729*p2t23) + ( . 18538*pzt24) 
compute f sl24=- ( . 03557*pztl) - ( . 08359*pzt2) - ( . 02880*pzt3) -( . 07783*pzt4) 

-(.09318*pzt5)-(.00593*p2c6)-(.08556*p2t7)-(.07281*p2t8)-(.01840*p2t9) 
-(.05343*p2tl0)+(.06830*p2tll)-(.10645*p2tl2)-(.14094*pztl3) 
+(.33244*pztl4)+(.32409*pztl5)+(.24912*pztl6)+(.33842*pztl7) 
+(.17023*pztl8)+(.18221*p2tl9)+(.07819*p2t20)-(.03113*pzt21) 
+(.08093*pzt22) -(.00647*pzt23)+(.03 025*pzt24) 
variable labels 

grp 1 weights factor I' 
grp 1 weights factor II' 
grp 1 weights factor III' 
grp 1 weights factor IV' 
grp 1 weights factor I' 
grp 1 weights factor II' 
grp 1 weights factor III' 
grp 1 weights factor IV' 
grp 2 weights factor I' 
grp 2 weights factor II' 
grp 2 weights factor III' 
grp 2 weights factor IV' 
grp 2 weights factor I' 
grp 2 weights factor II' 
grp 2 weights factor III' 
- - 2 weights factor IV' 

correlations variables=f siii to fsl24/statistics=descriptives 
subtitle '9 canonical invariance analysis' 

compute critlll- ( .90048*pztl) +( . 09741*pzt2) +( .05615*pzt3)+( .08840*pzt4) 
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fslll 


'grp 


1 


data 


fsll2 


•grp 
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data 


fsll3 


'grp 
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data 


fsll4 


• grp 
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data 


fs211 


•grp 


2 


data 


fs212 


'grp 


2 


data 


fs213 


•grp 
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data 


fs214 


'grp 
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data 


fs22X 


'grp 
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data 


fs222 


'grp 
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data 


fs223 


•grp 
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data 


fs224 


'grp 
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data 


fsl2l 


•grp 
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data 


fsl22 


'grp 
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data 


fsl23 


•grp 
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data 


fsl24 


•grp 


1 


data 
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compute critll2- ( . 50911*p2tl) - ( . 68916*pzt2) +( . 33707*p2t3) -( . 69424*p2t4) 
compute predlll=-( . 31370*p2t5) + ( . 50778*pzt6) - ( . 46168*pzt7) +( . 50972*pzt8) 
+(.77965*pzt9) 

compute predll2-(l. 16044*p2t5) +( . 60716*pzt6) +( . 09385*p^t7) -( .75882*pzt8) 
-(.72138*pzt9) 

compute critl21=« ( . 34770*pztl) + ( . 09867*pzt2) + ( . 35597*p2t3) +( . 52233*pzt4) 
compute critl22=— (.09842*pztl) + (1.09157*pzt2)-(.23021*p2t3) 
-(.36428*pzt4) 

compute predl21» ( . 42959*pzt5) +( . 31092*pzt6) - ( . 19267*pzt7) +( . 51866*pzt8) 
+(.09882*p2t9) 

compute predl22« (1. 09361*pzt5) +( . 66983*pzt6) - ( . 37908*pzt7) -( . 89606*pzt8) 
-(.55634*pzt9) 

compute crit211= ( . 90048*g2tl) + ( . 09741*gzt2) + ( . 05615*gzt3) + ( . 08840*gzt4) 
compute crit212= ( . 50911*g2tl) - ( . 689l6*gzt2) + ( . 33707*gzt3) -( . 69424 *gzt4) 
compute pred211=- ( . 313 70*gzt5) + ( . 50778*gzt6) - ( . 46168*gzt7) + ( . 50972*gzt8) 
+(.77965*gzt9) 

compute pred212=(l. 16044 *gzt5) + ( . 60716*gzt6) + ( . 09385*gzt7) - ( . 75882*gzt8) 
-(.72138*g2t9) 

compute crit221= ( . 34770*gztl) + ( . 09867*g2t2) +( . 35597*gzt3) +( . 52233 *g2t4) 
compute crit222=-( .09842*gztl) + (1.09157 *gzt2) -( .23021*gzt3) 
-(.36428*gzt4) 

compute pred221= ( . 42959*gzt5) + ( . 31092*g2t6) - ( . 19267*gzt7) +( . 51866*g2t8) 
+(.09882*g2t9) 

compute pred222=(1.093 61*gzt5)+(.66983*gzt6)-(.37908*g2t7)-( .89606*g2t8) 

-(.55634*g2t9) 
variable labels 



critlll 


•criterion 


grp 
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data 


grp 
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weights 


function 


1« 


critll2 


•criterion 


grp 
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data 


grp 
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weights 


function 
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crltl21 


•criterion 


grp 


1 


data 


grp 


2 


weights 


function 


1^ 


critl22 


•criterion 


grp 


1 


data 


grp 


2 


weights 


function 


2' 


predlll 


•predictor 


grp 


1 


data 


grp 


1 


weights 


function 


1« 


predll2 


•predictor 


grp 


1 


data 


grp 


1 


weights 


function 


2' 


predl21 


•predictor 


grp 


1 


data 


grp 


2 


weights 


function 


1^ 


predl22 


•predictor 


grp 


1 


data 


grp 


2 


weights 


function 


2^ 


crit211 


•criterion 


grp 


2 


data 


grp 


1 


weights 


function 


1^ 


crit212 


•criterion 


grp 


2 


data 


grp 


1 


weights 


function 


1^ 


crit221 


•criterion 


grp 


2 


data 


grp 


2 


weights 


function 


1' 


crit222 


•criterion 


grp 


2 


data 


grp 


2 


weights 


function 


2' 


pred211 


•predictor 


grp 


2 


data 


grp 
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weights 


function 


1» 


pred212 


'predictor 


grp 
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data 


grp 


1 


weights 


function 


1' 


pred22l 


•predictor 


grp 
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data 


grp 
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weights 


function 


!• 


pred222 


•predictor 


grp 
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data 


grp 
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weights 


function 


2' 



correlations variables=critlll to pred222/statistics=descriptives 



Note. Lower case commands were inserted after an initial run was conducted to 
obtain the values required in the second and last run. 



